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I. INTRODUCTION 

In recent years, some of the most ambitious charter school operators, with the support of 
philanthropic investors, have sought to increase the scale and scope of their work by creating charter 
management organizations (CMOs) that aim to replicate effective charter school models across 
multiple campuses. CMOs are nonprofit organizations with unified management teams that have 
operational responsibility for delivering the educational program and supervising the school leaders 
for groups of charter schools. CMOs seek to increase the number of students that have access to 
high quality educational options; to ensure consistently high educational quality across affiliated 
charter schools; and to promote improved educational practices in other public schools, both by 
creating models of effective instructional systems and by producing healthy competitive pressure. 
CMOs aim to achieve these goals while avoiding the bureaucratic rigidities and political conflicts that 
can be found in conventional public school systems. 

CMOs are now sufficiently mature to permit a systematic evaluation of their effectiveness. In 
New York City, we have access to a comprehensive data set that provides an opportunity to 
examine the impact of two established CMOs, Achievement First and Uncommon Schools (referred 
to hereafter as AF and Uncommon, respectively). In this report, we present findings from a 
nonexperimental analysis in which we estimate the effect of five New York City middle schools 
affiliated with AF and Uncommon on the achievement of their students. 

This report draws on the data collected for the National Study of CMO Effectiveness, which is 
being conducted by Mathematica Policy Research and the University of Washington’s Center for 
Reinventing Public Education. The NewSchools Venture Fund commissioned the study, which is 
funded by the Bill & Melinda Gates Foundation and the Walton Family Foundation. The national 
study is examining the practices and structures of a larger set of CMOs as well as CMO impacts on 
student achievement using both experimental and quasi-experimental methods. An interim report 
exploring CMO practices and structures has been completed (Lake et al. 2010); the final study 
report, which will include impact findings, will be released in summer 2011. 

The focus of this analysis is the achievement impacts of New York City middle schools 
operated by AF and Uncommon. The five schools examined here were included because they (1) 
were in a jurisdiction (New York City) in which we had data available; (2) are schools for which 
achievement data can be examined prior to entry (in third and fourth grades) and for each year of 
enrollment; and (3) have been operating long enough to include at least one year of outcome data 
for at least one cohort of students. The estimated impacts cover school years from 2005-2006 
through 2007-2008. We compare alternate methodological approaches to ensure the robustness of 
estimated impacts. The analyses we present in this report include adjustments to minimize potential 
biases caused by student selection, grade repetition, and attrition during middle school. 

A. Placing this Study in Context 

The existing empirical literature on the academic impacts of CMOs is virtually nonexistent, 
largely because most CMOs are very new. Mathematica’s ongoing national evaluation of CMOs aims 
to fill this gap with nationwide estimates on CMO impacts, to be produced in a public report in the 
summer of 2011. The literature on the achievement effects of charter schools in general is now 
extensive, but it has not offered any definitive conclusions. Findings are inconsistent, perhaps due to 
real variation in the performance of charter schools in different locations, and perhaps due to 
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methodological differences in studies (see, for example, Zimmer et al. 2009; Center for Research on 
Education Outcomes [CREDO] 2009; Sass 2006; Bifulco and Ladd 2006; Abdulkadiroglu et al. 
2009; Hoxby et al. 2009). For purposes of the current report, the most relevant prior study was 
released by Caroline Hoxby and coauthors in 2009, using admissions lotteries of New York City 
charter schools to estimate achievement impacts. Hoxby et al. (2009) found significant positive 
impacts of New York City’s charter schools, but they did not report results for individual charter 
schools or for charter schools affiliated with particular CMOs. CREDO (2010) similarly examined 
charter schools in New York City using matched analysis with virtual comparison students and also 
found significant positive impacts for charter schools. 

This report applies the most rigorous nonexperimental methods available to evaluate the impact 
of five New York City CMO middle schools operated by AF and Uncommon on the achievement 
outcomes of their students. 
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II. CHARACTERISTICS OF STUDENTS ENROLLED IN STUDY SCHOOLS 

In this section we describe the type of students who attend AF and Uncommon middle schools 
in New York City, including their demographic characteristics, poverty levels, special education 
status, and test scores prior to enrolling in middle school. We explore whether AF and Uncommon 
schools are serving a typical population by conducting two sets of descriptive comparisons. First, we 
compare the demographic characteristics and elementary school test scores of AF/Uncommon 
middle school students with their district counterparts. Then we examine how the grade repetition 
and attrition patterns of AF/Uncommon students vary from the patterns of other public school 
students. 

A. Data and Samples 

All of the data obtained from the New York City Department of Education was deidentified; 
each student received a unique identifier code to permit longitudinal analyses. The data set included 
the following variables: reading and mathematics test scores (middle school scores are the primary 
outcome in our analysis and elementary school scores are used as a baseline covariate), demographic 
characteristics (used as baseline covariates), and schools attended (used to measure student exposure 
to AF/Uncommon). 

Table II. 1 describes the five schools in our sample, the year each school opened, and the 
number of cohorts within each school that were included in the study. Due to data limitations (we 
currently have data only through the spring of 2008), the cohorts in the study were followed for one 
to three years, meaning that the earliest cohort was observed through seventh grade only. 

B. Characteristics of Students in Sampled Schools 

To investigate how AF/Uncommon middle school students might differ from other New York 
City public school students, we examined the fourth grade characteristics of future AF/Uncommon 
students and non-AF/Uncommon students. We compared AF/Uncommon students first with the 
district-wide student population and then with students observed at the subset of district elementary 
schools attended by AF/Uncommon students (“feeder schools”). The first comparison enabled us 
to identify baseline differences between AF/Uncommon students and non-AF/Uncommon 
students in New York City; the latter comparison was a means of investigating whether 
AF/Uncommon schools in New York City attract a different type of student within the elementary 
schools from which AF/Uncommon schools draw their student population. 



Table 11.1. Sampled Schools and Cohorts 



School (Year Opened) 


Cohorts in Study 


Achievement First Bushwick Middle School (2007) 
Achievement First Crown Fleights Middle School (2005) 
Achievement First Endeavor Middle School (2006) 
Uncommon Kings Collegiate Charter School (2007) 
Uncommon Williamsburg Collegiate Charter School (2005) 


1 (2007-2008) 

3 (2005-2006 to 2007-2008) 
2 (2006-2007 to 2007-2008) 
1 (2007-2008) 

3 (2005-2006 to 2007-2008) 


Note: A "cohort" is defined as the group of students who first enrolled in a CMO middle school's 

minimum offered grade (fifth grade) at the beginning of that school year. 
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First, we examined the demographic composition of the AF/Uncommon middle school 
students in our sample. These characteristics, which include gender, race/ethnicity, special education 
status, and limited English proficiency (LEP), are measured in fourth grade, prior to entry in an 
AF/Uncommon middle school. 1 (See Appendix Table A.l for descriptive statistics on these 
characteristics.) 

AF/Uncommon middle schools in New York City enroll a high concentration of racial 
minorities. Indeed, black and Hispanic students constitute nearly the entire population of students 
enrolled in AF/Uncommon middle schools (98 percent). Moreover, the proportions of both black 
and Hispanic students enrolled in AF/Uncommon middle schools are significantly 2 larger than those 
enrolled in other schools in New York City. AF/Uncommon students are also more likely to be 
black or Hispanic than students at their elementary schools who did not go on to attend 
AF/Uncommon middle schools. 

The baseline proportion of students in special education or with LEP is significantly lower at 
this sample of AF/Uncommon middle schools compared with both district- wide levels and levels 
observed in feeder elementary schools. (A detailed table of average special education and LEP levels 
at AF/Uncommon schools, feeder elementary schools, and host districts can be found in Appendix 
Table A.L) 

We also examined the baseline test scores of AF/Uncommon middle school students compared 
with New York City and feeder elementary schools in the district (see Appendix Table A.l). Relative 
to traditional public school students in the district, AF/Uncommon students had similar levels of 
achievement on the baseline reading test. In terms of the baseline mathematics test, however, 
AF/Uncommon middle schools in New York City tend to enroll students with significantly lower 
scores. Restricting the comparison group to students attending the same elementary schools as 
AF/Uncommon students indicates that AF/Uncommon students are much more similar to their 
peers in those schools than they are to other students in the district. The baseline test scores in both 
mathematics and reading reveal no significant differences between the elementary school 
achievement levels of AF/Uncommon and non- AF/Uncommon students in the feeder schools. 

C. Attrition and Grade Repetition 

We examined attrition rates in our sample because attrition is a potential source of student 
selection out of CMO schools that could bias our impact estimates. If an AF/Uncommon middle 
school does not retain a large portion of each entering student cohort through all grades it serves, it 
is possible that lower-performing students might exit at a higher rate, and the estimates of the 
school’s total effect may be overstated. Among the AF/Uncommon middle schools in the sample, 
the attrition rate was low. 3 Direct comparisons of attrition rates of AF/Uncommon schools and 



1 We do not report descriptive statistics on Free and Reduced Price Lunch (FRPL) eligibility because the FRPL 
indicator is missing for a substantial number of students in the New York City data set. As such, we think that the 
percentages of students eligible for FRPL may be underestimated and should not be compared to the rates of FRPL 
eligibility estimated in other studies. 

2 Throughout this chapter, all differences described as significant represent disparities with t-test y>-values less than 
0.05. All t-test significance calculations are two-tailed. 

3 To compare attrition levels at AF/Uncommon middle schools with those at other public schools, we defined 
attrition as follows: school transfers (either within or outside of the district) occurring during or immediately after each 
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conventional public schools in New York City are difficult because conventional middle schools in 
New York typically begin at grade six rather than grade five. We can, however, compare attrition 
rates between grades six and seven. During that period, traditional public schools record an attrition 
rate of 8 percent, compared with an attrition rate of 5 percent at AF/Uncommon middle schools. 

Finally, as part of our descriptive analyses, we examined the rates of grade repetition among the 
middle school students in our sample. Comparisons between AF/Uncommon school repetition 
rates and district-wide repetition rates are reported separately by grade in Table II.2. 



Table 11.2. Grade Repetition Rates by Grade Level 



Grade 5 Repetition Rates 


Grade 6 Repetition Rates 


AF/ Uncommon 


District 


AF/ Uncommon 


District 


(1) 


(2) 


(3) 


(4) 


0.08 


0.03** 


0.03 


0.02* 


N = 696 


N = 208,902 


N = 366 


N = 207,517 



Notes: Grade repetition represents the average proportion of each grade's students who will be 

retained in the same grade the following year. Grade repetition rates for AF/Uncommon schools 
are reported in columns 1 and 3. We report grade repetition rates only in grades five and six 
because the first cohort of students is observed for only one year in grade seven. 

* Difference from AF/Uncommon rate is statistically significant at the .05 level. 

** Difference from AF / Uncommon rate is statistically significant at the .01 level. 

As shown in Table II.2, AF/Uncommon middle schools in New York City retain students in 
fifth and sixth grades 4 at a significantly higher rate than traditional public schools in the district. This 
difference is especially evident in grade five, in which AF/Uncommon middle schools first enroll 
students who had attended non- AF/Uncommon public schools. Differences in grade repetition 
rates make it challenging to compare the achievement of AF/Uncommon students with district 
students. We discuss our approach to addressing this issue in the next section of this report. 



(continued) 

grade offered, up until the middle school’s maximum grade. (Each middle school was analyzed using its specific grade 
range, allowing the attrition analysis to disregard school transfers caused by a normal grade progression.) To compare 
attrition levels at AF/Uncommon middle schools with non-AF/Uncommon middle schools, we calculated the 
cumulative probability that a student entering his or her school in fifth grade will leave that school before completing 
seventh grade. (Given the constraints of the data, we observed attrition only in grades five and six.) These calculations 
account for the fact that at the time the data collection period ended, not all sample members had progressed through 
seventh grade. 

4 Due to data constraints, we could calculate grade repetition rates only in fifth and sixth grades. 
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